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ABSTRACT 

This paper addresses the main issues related to 
concepts of school effectiveness and their implications, with a focus 
on the selection of s chool “performance indicators. School 
effectiveness has many definitions, which is in part due to the 
different demands made by various social groups. The boundaries of 
performance within which schools should operate are generally 
ambiguous, but in essence have two strands: one that promotes social 
equity in terms of academic achievement, and one that promotes social 
competencies required for citizenship. The issues of pupil 
assessment, school evaluation, learning environment, and potential 
pr ob 1 ems with the use of performance i ndi cators are discussed. 
Formative models in performance-indicator development are evaluated 
in terms of their theoretical and practical uses. It is recommended 
that researchers categorize groupings of schools according to 
academic performance and socioeconomic features in order to identify 
common patterns and compare those groupings with other categories of 
schools. Four figures are included. (Contains 53 references.) 
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Abstract. 



The aim of this paper was primarily to address the main issues 
relating to concept of school effectiveness and their implications 
with regard to the selection of school performance indicators. School 
effectiveness has numerous definitions and this in part due the 
differing demands from the varir s social groups which make up 
our pluralistic society. The boundaries of performance within which 
schools should operate are generally nebulous, but in essence have 
two strands, promoting social equity in terms of academic 
achievement and the promotion of social competencies required for 
citizenship. Issues such as pupil assessment, school evaluation, 
learning environment, potential problems with the use of 
performance indicators are discussed. Formative models in 
performance indicator development are evaluated in terms of their 
theoretical and practical usefulness. 
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School Performance Indicators 
and School Effectiveness: 

the Co\:ceptions and the Critiques. 



1.0 



Introduction. 



Issue 1: School Evaluation and Performance Indicators have 
always been embedded within the political status quo. 
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BOX 1; TAKEN FROM THE MONTHLY PAPER OF THE NATIONAL SOCIETY, APRIL 1867. 



1.1: As illustrated by the content of Box One, the issues of school performance indicators are 

nothing new. The feelings of 'Fair Play' will strike a chord with teachers and others that 
remember the introduction of the 'League Table' to British Education during the 1980s. The State 
has always monitored the performance of State maintained schools, but the mechanisms by 
which it obtains, compares and disseminate information are realistically determined by political 
expediency. To some extent, information about the delivery of education in this country has 
been subjected to the same security safeguards that Whitehall should only reserve for 
diplomatic and military secrets. 

1.2: Educationalists (this include^ teachers, but to a lesser extent) have developed their own 

language in wliich they plan and operate, like most professions, that they can use if necessary, 
to protect their interests from the unwelcome intrusion of the general public. The opening-up of 
education to public scrutiny is something akin the processes simmering away in China under 
the watchful gaze of the old guard in Peking. In contrast to the foreground of Citizen Charter, 
there is a wide diversity of institutional activities in the background that perpetuate either a 
provider-driven or recipent-orientated model of education provision witliin individual schools. 
It is nearly impossible, for individual communities to engage in constructive criticism of their 
schools, as useful information for a democratic discussion is jealously guarded by policy 
makers. By reference to community it is inferred that there is a larger grouping of users of 
schools than parents and their children - community refers to all sectors of society who draw 
benefit from the education system and their entitlement to such benefits. This, of course, is 
translated into the notion of accountability, which came into usage during the great education 
debate of the 1970s. Sallis (1988) provides an interesting account of this period, but more 
paternally notes that 'accountability' was and is still is to some extent, perceived as thi'eatening, 
uncomfortable and as a devaluation of professional integrity by teachers. 



Design of the instructional environment, including actions that can be taken or conditions that can be 
impicmenled lo transform the initial state. 

Assessment of the transformation of the state that results from such action, i.e., assessment of specific 
instructional effects. 



1.3: As imperfect as they are, the 'League Tables' have provided the first window through 

which communities can observe what is happening in their schools in terms of academic 
outcomes. The choice of indicators has provoked a fiery debate which underlines the need for 
sensitive, some would say well intentioned indicators, that reliably project what quality of 
education pupils are receiving from the institution they attend. It should be noted at this 
juncture, that there is, as of yet, no holistic theory of education that genuinely encapsulates the 
activity of schools and the various environmental spheres that exert gravity on students' 




Issue 2: No theory exists that allows the objective selection of School Pis. 



Dfl 



1 



Box 2 



1. Description of the state of knowledge to be achieved through analysis of the task properties of a knowledge 
domain 



2. Description of the initial state with which one begins instruction by diagnosing characteristics. 



5. Evaluation of the attainment of the desired terminal state. 



(Taken from Cooley and Lohnes, 1976, p. 185) 




11 



3 



attainment to objectively remove the selection of school performance indicators from politicians 
impassioned with their own agenda. Kogan (1986) refers to this as the positivist dream. A 
framework for the development of such a theory is provided by Glaser and Resnick (1972), see 
Box Two, but this fails to take into account the impact of environmental and constitutional 
factors. Environmental factors are definable as the actual physical and economic environment: 
whereas, constitutional factors encompass two groupings of factors: the first, being the abilities 
of the pupil body and the professional skills of the teachers and the second, the beliefs, values 
and needs of different social groupings within a school’s immediate community and nationally, 
society as a whole. Yet in some respects a theory based solely on pedagogical processes might 
be the ideal foundation for the development of performance indicators, as it would 
undoubtedly emphasise the importance of good teaching techniques and whole school policies 
that benefit all pupils. 



Issue 3; Do league tables encourage a fiee-market in education? 




1.4: Central to the government philosophy for raising standards is parental choice and this 

was clearly demonstrated in the 1980s Education Acts. The 1980 Education Acts give parents 
the statutory right to make 'placing requests' for schools outside their catchment areas, and 
required education authorities to take these requests into account. The Acts also changed the 
rules on the number of pupils schools were required to admit, which allowed LEAs to fix the 
admission rate anywhere above 80% of the 1979-80 intake in response to falling roles 
(McClure, 1988). The result was that popular schools were disproportionately less affected and 
were less likely to become candidates for closure, or amalgamation (Willms, 1992). Thus, 
education was introduced to the rigours of a phantom free-market: one in which parents would 
actively seek the 'best' school for their children. This would effect schools to provide higher 
standards of education to gain extra market share. In a genuine free market organisations 
emerge in response to the needs of society, evolve and expands as it meets those need and 
those needs becomes more prevalent, to eventually stagnate and disappear when it can no 
longer competitively fulfil those needs, or when those needs no longer exists. Schools have 
tended to come into being as the result of strategic planning in response to the static needs of 
society, namely economic and social survival. Schools only close because they is insufficient 
pupils to support them and there is a readily accessible adjacent school where to decamp its 
pupils. The development of a genuine 'free-market' in education is further impeded by parents 
choosing schools for pragmatic reasons like proximity to the family home (Alder and Raab, 
1988), or because they perceived their child would be happy at a particular school (primary, 
but this might also apply to secondary) (Fetch, 1986). 

1.5: In some respects the 'League Tables' were designed to counter some of the in-built inertia 

in secondary education against a genuine 'free-market', in that they provide crude comparative 
data to assist parents in selecting a school for their son, or daughter. However, for the 
'educational literate' (see Harlen, 1994) they are of little use in their search for quality 
education for their child, as Barber (1994) elegantly put it:- 

Thc real problem xuith league tables is not that they are misleading but that they are 
insufficient. A matter as complex as the academic performance of a school cannot be 
summed up in one number. This is absurd as the moment in the Hitch-Hiker's Guide to the 
Galaxy when it zoas discovered the meaning of life is 42. 

(Michael Barber, The Guardian. 3 May 1994) 




1.6: The purpose of school performance indicators broadly falls into two distinct components. 

The first being a summation of a school as being either good or bad, or somewhere on the 
continuum in between. The second and more significant, is to measure the impact of 
implementing new school policies and where appropriate data was maintained to evaluate the 
consequence of current policy against some appropriate historical benchmark. At a more 
fundamental level performance indicators give a ’snap shot’ of whether a given school is 
meeting the criteria laid down by policy makers. They are best seen in the context of evaluation 
models. Richard's (1988) has developed a tri-part classification of American school evaluation 
models, with performance based models being only one of these classes. Richard's classification 
is as follows:- 

• Compliance monitoring models: Determine if schools are operating in accordance with 

some predetermined, externally exposed standard (s) 
emphasising "inputs" (i.e., resources available) rather 
than performance. 

• Diagnostic monitoring models: Focus on student improvement and, therefore, mostly 

rely on criterion-referenced pre- and post-testing 
patterns. They are primarily formative in nature and 
focus on how assessed student needs are being 
addressed. 

• Performance monitoring models: Focus primarily on norm-referenced achievement tests 

given usually at the 4th, 8th and 10th grades. They are 
summative and intended to enable schools and districts 
to be compared currently and longitudinally. 

(Taken from Gallegos, 1994, pp.45-46.) 

1.7: All tliree model groupings draw their legitimacy from the interests of different stake- 

holders (please refer to Figure 1) in the education system and in some ways this reflect the 
hierarchy of control within the system; compliance models fitting the needs of central 
government; diagnostic models the needs of schools. Performance models best fit the needs of 
parents, but also provide another mechanism of control for central government. No one 
grouping is essentially more valid, rather each grouping gains additional legitimacy when used 
in conjunction with each other. 




FIGURE 1: STAKEHOLDER MAP OF A SECONDARY SCHOOL (TAKEN 
FROM ASPINWALL, SIMKINS, WILKINSON AND McAULEY, 1992, p.S6). 



Note; The length of the arrows indicates the perceived importance of each stakeholder group: the shorter ihe arrow, 
the greater the perceived importance. 



2.0 What construes performance within a school setting. 



Issue 5: The construct of an effective school is NEBULOUS! 



2.1: Schools carry a wide remit of responsibility, which has been vested in them by society. 

The purpose here is not to question the legitimacy of these roles, or for that matter their 
relevance, but to acknowledge their existence. The roles of schools might be widely described 
as:- 

• The transfer of Knowledge, Skills and Understanding (KSUs) from one generation to the 
next that are essential for each child's integration, via adulthood and citizenship, into 
society. 

• Encouragement of self-discipline, self-esteem and confidence within each child to ensure 
they can fully participate in and contribute to society. 

• The intellectual, emotional, moral, aesthetic and physical development of each child. 

• To introduce to each child ideas and opportunities that might not be available in its 
immediate neighbourhood. 

• To identify each child's strengths, educational needs and career interests and as far as 
possible, within the constraints of the National Curriculum and the school's resources, 
tailor a programme of education (combination of subjects) appropriate for that child to 
obtain employment. 

• The transmission of, as far as possible, a common value system that can act as a social 
glue. 

2.2: Just as society is made up of various interest groups of differing political hues, the 

interpretation of the above into action is both heterogeneous and contentious. The matter is 
further complicated by some interest groups adding and subtracting functions within which 
schools should operate. Schools as evidenced by the above do not operate witliin the context of 
one dimension of performance and indicators used to estimate the relative (normative) 
performance of a school should take as many of these dimensions into account: therefore, 
schools genuinely work within nebulous boundaries of performance. It is against this context 
that performance indicators need to be developed that are both transparent and meaningful to 
all stakeholders in the education system, and are also an expression of more than one of the 
above principles. This would enable an evaluation of schools' adherence to the implicit social 
contract that they hold with their local community. 



Issue 6: The Government has yet to articulate what is an effective school. 


\ 





2.3: Brighouse (1989) neatly puts into context the application of the above principles of 

schooling; he laments the lack of Government interest, as demonstrated in HMI and DES 
documents, in what constitutes an effective school. He noted that this omission might be partly 
explained by the diverse opinion about what is a good school and this may have resulted in 
elusiveness of the topic in HMSO publications. What literature there was at the time of writing 
his paper relied little on quantified evidence, preferring instead a subjective impressionistic vieiu of 
education. Given the fluid state of education this might be no bad thing. Brighouse refers to 
effective school as:- 



. . . can first and foremost, be recognised through its pupils, its staff and its community. 

Recognition is [best seen as] parents zoho say 'My child simply cannot wait to go to 
school This should hold higher precedence than official and media recognition. 

It xuill be from the Kitchen staff or cleaners zoho comment, 'It is alright up at Bluebells. 
Their head is a good sport. I go therefor the people and not the money and I zoouldn't miss 
it for the zoorld.' 

Or local employers stating something like: We'd alzoays take one from St Thomas. They 
alzoays seem to produce such zuilling and confident youngsters.' 

Litmus tests of outstanding schools, therefore, are not just public occasions or examination 
results but also and importantly private zoitness. 



I Issue 7: The best judges of school effectiveness might be the support staff! 

2.4: By private witness, Brighouse is referring to non-teaching staff and others, as they can 

form a more objective assessment: who can tell the depth and quality of relationships within 
schools and can readily see whether the school truly celebrates all its constitutes. They note the 
consistency of the school towards pupils, pupils' personal development and whether pupils are 
unafraid, free, self-disciplined and autonomous. However, there may be evidence that SMTs 
attempt to silence, if not 'brain wash' their support staff, as demonshated by the no-talking to 
non-approved newspaper clause in contacts and the various obligatory INSETs. 




I Issue 8: It is easier to spot a good school than to define school effectiveness! 

2.5: Brighouse is essentially referring to the human aspect of the education system. He is in 

some respects debunking J:he whole concept of a school being a cohesive unit (and possibly also 
social class) and this is refreshing in that many of the conceptual models of educational 
attainment and consequent school effectiveness relegate the pupil from being a free thinking 
individual with some, maybe limited, autonomy over their own destiny to a mere mindless 
pawn within the power play of educationalists and politicians. Brighouse sums up effective 
schools as:- 

• Having shared values: effective schools know where they stand on the key issues like 
race, equality of the sexes, the family, prejudice and educational philosophy and this will 
have resulted form exhaustive debates. 

• Self-evaluate within the context of common school purposes which depend on 
interdependence and collegiality. 

• Has a set of principles (see Box Three for those Brighouse suggests). 

• Leadership. 




Issue 9: The leadership quality of the Head impacts on the effectiveness 
of a school, but so does political infighting of the SMT. 



2.6: Educational leadership has probably received the most attention of the four and this may 

reflect the conveyor belt of studies that was triggered by mostly the American military in the 
1940s and the pursuing interest of industrial and commercial corporations in what constitutes a 
good manager. It was no big step to transplant and develop these ideas within the context of 
education. Wright and Taylor (1986) argued that they are six major styles of interaction between 
managers and staff which fall onto the autocratic-democratic continuum (Teir, Tell and Sell', 
Tell and Listen', 'Ask and Tell', 'Problem Solving' and 'Ask and Listen') and it is suggested that 
some of these styles may be appropriate for different events and processes within a 
organisation. Brighouse simply defines three categories of educational leadership from the 
literature and these are as follows:- 

• Perceptive Professional Developers 

• System Maintainers 

• Inadequates 

2.7; More colourful definitions by Baddcley and James (1987) might be more appropriate, as 
these reflect the political constituents and grabs for control of the school agenda by SMT 
members (i.e., the Fox, the Owl, the Donkey and the Sheep). As brilliant as it is, the Brighouse 
paper fails to consider the pivotal role of subject co-ordinators and in some schools the power 
devolved to department heads to pursue their own interpretation of whole school policies. May 
be his catch all statement, 'In some strange way the sum of the parts of an effective school is exceeded by 
the totality of what it stands for' (p.l41), suggests that he was possibly aware of this at the time of 
writing his paper; as this statement implies that ineffective schools are those where one or more 
the teaching departments' policies are failing their pupils. This might even undermine the 
legitimacy of whole school performance indicators: as I have argued elsewhere (November 1993 
and January 1994) that it is more legitimate to assess performance at departmental level. 



Box 3 

1. Children should be treated as they might become rather than as they are, 

2. all pupils should be equally valued; 

3. teachers should have the expectation that all their students have it in them to walk a step or two with genius, if 
only they could identify the key to unlock; 

4. the staff unitedly stand for the successful education for the whole person; 

5. the staff contribute to the development of mature adults for whom education is a lifelong process and propose 
to judge their success by their pupils' subsequent love of education; 

6. the staff should try to heal rather than to increase diversities, to encourage a self-discipline, a lively activity to 
breed lively minds and good health, a sense of interdependence and community. 
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Issue 10; Equity or Excellence. 



3 



2.8: In a open letter to Parliament, Cox and Dyson (1969) wrote:- 

'You can have equality or equality of opportunity. You can't have both.' As Angus Maude 
demonstrated in our first Black Paper, equality of opportunity is totally different p-om the 
present cult of egalitarianism, which is indeed its chief enemy at the present time. The 
frightening aspect of egalitarianism is while it costs far more to bring into effect than 
equality of opportunity, it disintegrates the standards and structures on which education 
depends. It is a levelling doion process, actively unjust to brighter children, who become a 
nexo under-privileged, and for this reason dangerous for the nation as a whole. 

Cox and Dyson v/rote their paper during the introduction of comprehensive education to the 
English and Welsh education system. By the late 1960s grammar schools were being seen as a 
passport to more lucrative p'-ofessions within society. They aclrieved this by stretching the 
more able; however, those that did not pass the eleven-plus were relegated to less academically 
demanding secondary moderns, where teachers aspirations were towards their pupils filling 
the then still plentiful manual positions in society. This often resulted in a waste of vital talent 
(both meanings), as capable secondary modern pupils were not pushed towards GCE. The 
problem was compounded by the grammar schools being able to recruit and retain higher 
calibre teachers, thus concentrating the provision of good quality education. There can only be 
comparative excellence; the brilliance of excellence only exists when set against the mediocre. It 
is the creation of mediocre schools through various socio-political processes that ensures the 
existence of excellent schools. The crucial issue to be addressed by school performance 
indicators is whether pupils are being given equality of opportunity to succeed via good quality 
education. This is only achievable tlirough continual improvements in whole school policies 
and staff development programmes. Indices are desperately needed that quantify these 
processes, which can be validated and embedded within the education system. This is the only 
mechanism for raising upwards the quality of state education provision and complacency is 
likely to fuel, if not falling standards, perceptions about falling standards. 

2.9; Caldwell (1988) refers to a new conception of equity. In the nineteenth century, equity 
referred to "free, compulsory and secular" education for a!! children. The paradigm then shifted 
to equal allocation of resources ar.J this led to the introduction of comprehensives that was 
feared by Cox and Dyson. The most recent reconceptualisation is each student has access to 
more differentiated resources that best meet his or her needs, interests and aspirations rather 
than a fixed 'aggregated mix of resources'. The only mechanism for achieving this is through 
what Mann (1988) refers to as parental engineering, which is distinct from social engineering 
projects of governments during the 1960s and 1970s, such as Headstart. By parental 
engineering, Mann is referring to parents choosing schools that share their values and are able 
to provide specialised curricula in those areas that particularly interest their child, or provide a 
sound basis for that child to enter Iris or her (parents'?) chosen career. The influence of parental 
engineering is present in most societies and is jealously articulated, as demonstrated by a recent 
article in The Economist (17th June 1994) which reports some Californian parents resisting the 
introduction of standardised testing, akin to GCSE and SATs as this might corrupt the values 
systems of the schools they have chosen for their children and introduce said children to ideas 
that are contrary to their parents' belief systems. One definition of excellence might be does a 
school fulfil parental aspirations and it is true that schools that efficiently fulfil such aspirations 
are perceived as excellent. However, parental aspirations do not always match the needs and 
aspirations of society, and in some circumstances inhibit the intellectual and personal 
development of their child: where such inhibition occurs, it can not be argued that a school is an 
effective provider of quality education. 



2.10: Effective schools in summary do not only serve the needs of parents and teachers, cater 
for pupils of differing abilities, but also address positively the issues confronting their 
communities. Equity within this context is not only measurable in terms of absolutes like 
number of GCSE points, but also terms of social competences prerequisite for citizenship: 
citizenship being the ability to participate fully economically and socially in society. 
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3.0 



Assessment of student attainment: index of equity. ■ 




3.1: The recommendations of the Task Group on Assessment and Testing (TGAT) were that 

pupil assessment should emphasise criterion-referenced, fonnative assessment based on a 
progression of skills related to pupils* expected rate of educational development (Willms, op cite). 
Criterion referenced tests aim to describe what a pupil can do, within the context of a specific 
achievement domain. Criterion referenced tests depend on the definition of competencies or 
essential domains (KSUs) and these can be ordered hierarchically. Two excellent examples 
being GASP (Graded Assessment in Science Procedure) (see Swain, 1987) and SAIL (Staged 
Assessment in Literacy) (see Beveridge, 1991). In many ways the National Curriculum was 
meant to be criterion referenced, however, the descriptors or SoAs, were somewhat Vdgue, 
which added confusion to teachers overwhelmed by the sheer paperwork that results from any 
genuine criterion referenced procedures of assessment. Attempts to reword SoAs into more 
specifically observable competencies in the summer of 1991 resulted only in a more threatening 
array of documentation with each SoA broken down into its constituent meaning (Smith, 1993; 
personal communication). 




3.2: The issues of criterion (CRT) versus norm-referenced (NRT) testing are extensively 

covered by Anastasi (1988) and Kline (1993). The salient points in regards to school 
evaluation are:- 

1. CRTs by their very nature are restricted to one, or few tasks, which are assessed 
formatively: they provide a mechanism for identifying and alleviating pupils' difficulties, 
subject to the caveat of available teaching resources. 'Teachers tend to favour CRTs over 
standardised, nonn referenced tests (NRTs) because CRTs identify areas requiring further 
teaching, and provide infonnation on whether specific pupils have mastered a particular lesson.' 
(Willms, op cite, p.l7). 

2. Whereas, NRTs typically attempt to assess a larger cross section of the curriculum and 
produce a summative assessment of each child's attainment. The resulting data can be 
manipulated to provide rankings, place pupils in percentiles and generally provide a 
mechanism by which observers, and pupils themselves, can deduce each pupil's relative 
performance. 

3. Many parents want the results of NRTs so they judge their child's standing against their 
age-peers. 

4. Classic test theory dictates that the ability of a test to identify individual differences in 
pupil attainment increases with the number of test items. 

3.3: Essentially only NRT are usable for comparing schools on the basis of pupil attainment at 

this juncture in time and they do provide comparative data. Critical to their fairness is a 
common syllabus and pupils having an equal opportunity to succeed via high quality delivery 
of subject matter. Admittedly, this is not always the case and some schools are expert in training 
their pupils to pass exams with minimal understanding of the subject. This leads us back to the 
issue of indicators being able to demonstrate equality of opportunity to succeed. 



4.0 



Evaluation of Schools. 



Issue 13; What can we leam from the American system of school evaluation? 







4.1; The American model of school evaluation consists of agencies at state, regional and 
national level and has been ar. integral part of the American education system since the end of 
the nineteenth century, as a means of accrediting schools and was instituted to determine if 
funds allocated were being spent appropriately. The accreditation process has grown in 
complexity and is performed by agencies at all three levels; and has moved on from the 
purpose of ensuring compliance to regulations to include assessment focusing on school 
improvement and student outcomes. There is an immense diversity of agencies with differing 
procedures and to ensure consistency a Joint Committee was formed in the early 1980s. joint 
Committee Standards were drawn up for the evaluation of transition programmes (institutional 
strengthening of individual or clusters of schools), which have been instrumental to the 
development of improved evaluation of professional standards in education (see Nowakowski, 
1990; Gallegos, 1994). Parallels can drawn between the US system and the British system of 
evaluation, in that there is now and probably has always been in some LEAs two or three levels 
of agencies involved in school evaluation. The picture is further complicated by the ability of 
GMS schools to buy in their own evaluations; this makes an evaluation of the American System 
even more interesting. Joint Committee standards consist of four strands (a little like 
Attainment Targets) under which checklists are arranged, as shown in Box Four. 

4.2: Gallegos (1994) noted five common weaknesses of the available model when reviewing 

the fifty one available models for evaluating school performance in the United States. These are 
as follows:- 

• No clear indication that evaluators had received any significant training in school 
evaluation. 

• Little evidence that evaluations are based on any acceptable and recognised set of 
standards for evaluating programmes or products. 

• Limited evidence that a systematic approach to school evaluation is understood or 
practised. 

• Limited attempts to address issues of quality in any meaningful way. 

• No indication that a metaevaluation of school personnel evaluation practices was 
conducted or even suggested. 




4.3; Like the American system, school evaluation is fragmented in the United Kingdom. 
Essentially, there is the Ofsted which acts as an national body, LEA inspection teams and 
independent registered inspectors who report to the HMCI in accordance to the Education 
(Schools) Act 1992. The local government reorganisation in 1974 resulted in the establishment 
of larger LEAs and more coherently organised teams of educational advisors (sometimes called 
inspectors) (Wilcox, 1989) who could orchestrate more frequent school evaluations than the 
HMI. The Audit Commission occasional paper Losing an empire, finding a role: the LEA of the 
future defined six roles for LEAs to reflect the changes introduced by the Education Reform Act 
1988 ('Leader/ Visionary', 'Partner', 'Planner', 'Provider of information* which is closely linked 




to ’Regulator‘s and 'Banker'). The 'Regulator' role being school evaluation via inspection and 
collation of test data as an indicator of acadenaic attainment. There is considerable inconsistency 
in the methods used by LEAs, with only a few LEAs having substantial programmes of whole 
school inspections in place (Wilcox, Gray and Tranmer, 1993). The government's objective, as 
reported by DES (1991), is to have a radical, market driven approach based on independent 
registered inspector (IRI) teams (trained by the 'old guard' (i.e., those that remain from the 
HMI)) conducting full inspections on a four year circle. The radical component being overt 
monitoring of school performance. Yet when the four-year cycle will come into being is a crucial 
area of concern, in that it was stated a year later that:- 

At the prevailing rate of inspection by Her Majesty^s Inspectorate of Schools before the 
change introduced in the Education (Schools) Act 2992, it would have taken some sixty 
years to cover every secondary school in England, and hoo hundred years to inspect each of 
our tioenty thousand primary schools. 

(HMSO, 1992, p.8) 

4.4: IRls are trained in accordance with The Handbook for Inspection of Schools (Ofsted, 1992), 

which details the information that they must collect. In some respects the instructions within 
the manual are open to individual inspectors interpretation and with the exception of 'hard 
data' collected in the form a questionnaire administered to Headteachers, most of the data is 
both qualitative and subjective. The legitimacy of this data is dependent on the use of multiple 
sources. It would be entirely unrealistic to expect inspectors to be totally objective, but 
according to Judd (1994) there are expected to look for working combination of teaching 
methods and not to make judgement situated in the traditional-modern teaching debate. 
Wilcox et al (op cite) note that many LEAs model their approach to evaluating schools on that of 
the now defunct HMI. The sheer quantity of qualitative data being generated by both the LEAs 
and Ofsted serves to underline two fundamental weaknesses in the evaluation system: first, the 
need for more transparent Pis common to all assessment that would meet the utility criteria of 
the US joint Committee, thus enabling evaluation of schools at community level; and second, 
the need for a mechanism to convert this qualitative data into objective statistical indices that 
can be utilised for meta-analytical studies in school effectiveness. 



Issue 15: What is meant by school effectiveness? 



4.5: Mann {op cite) refers to 'effective school cosmology', whereby "...the label ^effective school 

has been pasted on a lot of situations that have little resemblance to the movement's original form' (p.4). 
Initially, effective schools were defined as having two distinguishing characteristics:- 

1. Effective schools raise student acliievement above that which would have been expected 
on the basis of family background. (The effective schools movement grew out of concern 
for the children of the American underclass.) 

2. An effective school demonstrates the following characteristics:- 

a. Strong, instructionally centred leadership; 

b. Positive teacher related behaviour; 

c. A secure organisational climate; 

d. Curricular artefacts that maximise instructional time that are aligned to ... 

e. A formative pupil progress evaluation system. 
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Box 4 

A) UTILITY: 



B. FLA31BILITY- 



C PROPRIEIY: 



D. ACCURACY: 



The utility standards are intended to ensure that an evaluation will serve the practical information needs of a given 
audience. 

1. Audience Identification. 

2. Evaluator Credibility. 

3. Information Scope and Selection. 

4. Valuational Interpretation. 

5. Report Clarity. 

6. Report Dissemination. 

7. Report Timeliness. 

8. Evaluation Impact. 

The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and fugal. 

1 . Practical Procedures. 

2. Political Viability. 

3. Cost Effectiveness. 

The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard 
for the welfare of those involved in the evaluation, as \% ell as those affected. 

1 . Formal Obligation. 

2. Conflict of Interest. 

3. Full and Frank Disclosure. 

4. Public Right to Know. 

5. Rights of Human Subjects. 

6. Human Interactions. 

7. Balanced Reporting. 

8. Fiscal Responsibility. 

The accuracy standard are intended to ensure that an evaluation w'ill reveal and convey technically adequate information 
about the features of the object being studied that determinate its worth or merit. 

1. Object Identification. 

2. Context Analysis, 

3. Described Purposes and Procedures. 

4. Defensible Information Sources. 

5 Valid Measurement, 

6. Reliable Measurement. 

7. Systematic Data Control. 

8. Analysis of Quantitative Data. 

9. Analysis of Qualitative Data. 

10. Justified Conclusions. 

1 1 . Objective Reporting. 



4.6: The five factors listed above, according to Mann, are and were, the most contentious 

aspects of the effective school movement, probably because they rest responsibility at the feet of 
the providers rather than blame the socio-economics realities of the child's home background. 
Various authors have recommended some form of decentralisation of decision making to 
schools as a mechanism for raising quality, as those who actually deliver the service, at least 
theoretically, should be able to make more appropriate decisions that directly influence their 
students. However, the flattening of the bureaucracies has resulted in more members lower 
down the (school) hierarchy becoming managers. Depending on one's perspective this can lead 
to job enlargement which is positively associated with lower stress levels; or role ambiguity 
and/or role conflict, which raises teachers' anxiety levels. Since stress is essentially Lhe 
interaction between anxiety and fatigue consequent from an increased workload, one might 
predict that, in the absence of support, 'decentralisation' will impact negatively rather than 
improve the effectiveness of individual staff member and collectively the school as an unit. 
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5.0 Learning Environment. 



Issue 16; The socio-environmental aspects of being a pupiL 



1 



5.1: The role of school inspectors is to assess the learning environment of the pupils and there 

is the need to move towards more pupil-centred indicators. Box Five paraphrases the 
conclusions drawn by D.C. Miller and W.H. Form on Mayo's Hawthorne studies about the 
influence of social groups on worker productivity, as reported in Kennedy (1992), and Box Six 
includes the conclusions drawn by Keeve (1972) review of the literature relating to affects of 
classroom environment on student motivation and industry. The direction of causation is 
unclear, as Beshers (1972) aptly puts it: 

Whether the peer-group effects are necessarily opposed to scholastic achievement; or 
luhether peer-group effects are essentially unrelated to the school system but perhaps create 
competitive goals; or whether peer group can be used to support the school system is an 
unclear area of social theory. 

(p.531) 

5.2: The socio-environmental aspects of being a pupil are undergoing a process of change and 

this is a reflection of changing social structures and the consequent needs of their school's local 
community. Posch (1994) identifies four challenges for schools that will influence the learning 
environment of pupils in years to come and these are: 

1. The negotiation of rules between staff and pupils in response to changing family 
structures. The definition of in loco parentis will become less authoritarian in nature 
reflecting the democratisation of the family. 

2. The lack of 'social continuity' within society that is essential for prosocial co-operative 
behaviour. Schools should therefore foster long-term relationship and discourage short- 
term relationships mediated by short-tern*, gain, therefore facilitating 'social continuity' 
within the school environment. 

3. The 'dynamic qualities' of modern life: the frontiers of state intervention over individual 
decision-making has been rolled back. Pupils should be provided with a framework 
within which they can model responsible and constructive decision making and to 
demonstrate that they do have a role in their communities. 

4. Encouraging 'reflection and critical approach to knowledge'. The growth in scientific 
knowledge has made the selection of important knowledge difficult and contentious and 
therefore pupils need to be encouraged to grasp the issues relating to scientific enquiry 
and to evaluate existing knowledge to derive their own understandings and opinions. 

5.3: Implicit from the above four statements is that the learning environment within schools is 

undergoing a period of transition, and it is feasible that it will be sometime before any level of 
stability is reached. The success of schools both within academic and social dimensions will be 
increasingly dependent on the efforts of both staff and pupils and their motivation to meet new 
challenges. Unfortunately, it can be argued that the aspirations and abilities of pupils to be free- 
thinking individuals are currently neglected in many conceptions of school effectiveness: this 
argument can also be said to be true for the teaching staff. Often criticisms against Pis rest on 
the assumption that teachers, like pupils, are only currency within the education system. 
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Box 5: 

1. School work is a group activity. 

2. The social world of the pupil is primarily patterned about school activity. 

3. The need for recognition, security and sense of belonging is more important in determining pupil's morale and 
productivity than the physical conditions under which her or she works. 

4. Pupils' complaints are not necessarily an objective recital of facts: they are symptoms manifesting disturbances 
of the individual's status position. 

5. The pupil is a person whose attitudes and effectiveness are conditioned by social demands from both inside 
and outside the school. 

6. Informal groups within the school exercise strong social controls over work habits and attitudes of individual 
pupil. 

7. The change from an established to an adaptive society ... tends to continually disrupt the social organisation of 
a school and education generally. 

8. Group collaboration does not occur by accident; it must be planned for and developed. If group collaboration 
is achieved, the work relations within a school may reach cohesion which resists the disrupting effects of an 
adaptive society. 



Box 6: 

1. Achici^ement press of the classroom: the extent to which teachers stressed the importance of academic achievement 
and the desirability of home study, and required high standards of achievement. 

2. Proihsion for independent study and enquiry: Tlie degree to which learning was directed by student interests and 
abilities, with student responsibility for planning units of class work, allowing several activities in progress in 
the classroom at one time. 

3. Emphasis on work habits and order: The extent to w'hich the teacher was engaged in responsible, businesslike, and 
systematic classroom behaviour. 

4. Affiliation and warmth in the classroom: A function of the number of statements used by the teacher, the 
occurrence of sympathetic laughter, the knowledge which the teacher had of the student's home background, 
and the links established by the teacher with each child and with the child's parents. 

5. Stimulation for lear7ung: The availability and use of a variety of instructional materials, the use by the teacher of 
a variety of activities, the display on the walls of the room of charts and items prepared by the students, and 
the use of stimulating assignments. 

6. Emphasis on oral work and correct use of language: Primarily a function of the extent to which students, as opposed 
to teachers, did the talking, and some of the kinds of responses which teachers made to the student discussion, 
especially with respect to the use of language. 

7. Academic guidance and instruction: A function of such aspects of teacher behaviour as the use of introductory 
and concluding comments in a given lesson, and the amount of teacher-student interaction concerned with 
instructional procedures such as homework and review. 

(Taken from Cooley and Lohnes, 1976, p. 183-4) 



6.0 Potential problems and professional issues relating to the 
use of performance indicators in the United Kingdom. 




6.1: Helsby and Saunders (1993) provide a qualified critique of the use of Pis in British 

education and trace their development back to F.W. Taylor - the application of scientific 
management for maximum efficiency - and Tyler (1949) who applied the behaviourist model or 
'objectives model' to teaching. Both Taylorism and Tylerism have been indicted with imposing a 
mechanistic approach to teaching that disablitates teachers' professional judgement. According 
to Helsby and Saunders these movements have introduced the metaphor of the production line 
to education - even though historically education has be bound up with the production of 
employable citizens and, in many ways, the products of education fall into fairly homogenous 
groups. They go on to critique the concept of accountability as defined by the industrial 
metaphor:- 

For the anxious to ensure accountability in education, the lure of the industrial metaphor is 
enticing, for it offers the comforting illusion that complex and costly educative processes 
can be reduced to the straightfonoard linear progression of a production line with tangible 
and easily measured end products. 

(p.61) 

Helsby and Saunders are correct in that professional judgement provides valuable insights and 
a synopsis of educational events, but professional judgement is in essence opinion; therefore, 
one can not agree with them that it provides an accurate assessment of education provision. 
This forces us to accept that the concept of quality education is essentially cliched, in that its '... 
(b)cauty is alzuays in the eye of the beholder'. Whilst the empiricist model is still on the ascendancy 
in educational research, and in many respects rightly so, to critique it on the grounds that it 
causes the removal of the humanistic approach to educational enquiry is denying that 
educational enquiry is only effective when it is pluralistic in approach. Citing Eisner's (1977) 
concept of educational connoissuership and Littler's (1978) definition of Taylorism - 'the 
bureaucratisation of the structure of control' and 'the creation of new social mechanisms for 
conshtuting effort standards within an accelerated dynamic of deskilling' - does not add 
credence to this argument. Educational connoissuerships like all connoissuership is based on 
indefinable and hazy perceptions of quality, relying on the use of metaphorical comparisons to 
transmit that something is good. Connoisseurs can easily tell you something is bad, but tend to 
go into raptures about how good something is, but not why! Littler's view is jaundiced to the 
extent, that it fails to accept that there are more effective patterns of working that employees can 
adopt, and that rigorous imposition of inflexible and inappropriate working methods that he 
suggests result from Taylorism are contrary to the original principles of scientific management. 
Carried to its logical conclusion, Littler definition of Taylorism would imply the purpose of 
training is to deskill. If this was true, then it could be argued that empiricist approach to 
educational effectiveness is flawed and inappropriate. 



Issue 18: The validity of Helsby and Saunder*s four criticisms of using Pis. 




6.2: Following their critique of the current trends in British education, namely the 

development of behaviourally defined and therefore observable attainments which can be 
demonstrated by pupils, Helsby and Saunders go on to critique use of Pis in education on four 
grounds: 



1. the use of performance indicators deprofessionalised teachers; , 

2. ' their use in partial and distorted evaluations; 

3. they do not work as indicators of performance because causal relations cannot be 
demonstrated; 

4. performance indicators have little educational or developmental value. 

With regards to point one, Helsby and Smith are arguing that Pis smack of 'Big Brother' and 
whilst it is true that CIPFA published Performance indicators in the education service: a consultation 
document in 1984, it does not necessarily mean that teachers actions are controlled by Pis as they 
suggest. To suggest that Pis will direct teachers' priorities is in many respects a devaluation of 
the professional integrity of individual teachers. 

Point two refers to referencing outcomes against predefined desired outcomes. Helsby and 
Saunders have two problems with this: the first, being that evaluation by objectives inevitably 
focuses the evaluation questions, often the expense of unintended effects of the educational 
policy. The second, being Pis prevent the adoption of a fluid curriculum and it is argued on the 
basis of Elliot’s (1991) work on action research for educational change that open-ended learning 
outcomes cannot logically encounter outcome indicators. There many outcome measures of the 
effectiveness of schooling and in many ways the drama of open-learning is a means to achieving 
one or more these outcomes. The first, assumes that the teacher doej not self-evaluate in situ. 

The third point deals with the measurement of phenomena of education and the application of 
common sense that no one manipulable factor wholly determines educational outcomes. The 
attribution of cause to any one factor is incorrect to the extent that it can only influence in 
conjunction with factors and their constituent variables the educational outcome of a student. 
This in no way undermines the validity of using such factors, or specifically the more influential 
factors, as performance indicators. 

Point four essentially deals with the definition and usage of performance indicators. Incorrect 
definition or usage invalidates performance indication. There is no reason why enabling 
indicators as Helsby and Saunders refers to them cannot be widely used in the educational 
context. 

In conclusion, criticisms of Pis deal more with their utilisation, but their incorrect usage does 
not invalidate them as a potential mechanism of raising educational quality. It is only within a 
coherent framework that they can be used effectively, such a framework would also provide a 
model within which to develop more appropriate Pis for the educational context. 



7.0 Models of school performance indicator development. 



i Issue 19: Process indicators * the way forward? 



7.1: The danger inherent in producing Pis is that one ends up with a disjointed corpse, 

whereby the indicators produced only describe parts of the organisation's effectiveness. It is 
therefore necessary to have a model within which to develop Pis that can act as a co-ordinating 
skeleton. Scheerens (1990, 1991) has successfully applied contingency theory to the context of 
schools, using what he refers to as a contextualised model of schooling (see Figure 2) to develop 
an integrated model of school effectiveness (see Figure 3). The validity of contingency theory 
witliin this context draws from two of its particular strengths (it could be argued that the 
introduction of the market metaphor to education, has in itself, necessitated the use of 
contingency theory within the context of education): first, the notion of effectiveness, unlike its 
definition in the majority of the literature where it is shrouded within the metaphor of 
productivity, is multifaceted, in that effectiveness can also mean:- 

• resource allocation; 

• stability and control in functioning; 

• cohesion and morale among organisational members, 

(Cameron and Whetton, 1983) 

7,2: Contingency theory acknowledges that effectiveness can mean different things to 

different organisations and similarly each organisation's definition of effectiveness changes 
according to its needs, its current stage of development and its environmental circumstances, 

7,3: Secondly, the application of contingency theory is also a recognition of the importance of 

external incentives on the development and adherence to an achievement oriented policy: "The 
political xuillfor a school to succeed is perhaps the most essential condition for actual school effectiveness" 
(Schereens, op cite, p.71). Figure four attempts to demonstrate what might be the dominant form 
of relationship between schools and their stakeholders; whereby communication is mediated 
solely through the use of rewards (praise, extra resources from local and national government 
and commerce, parents selecting a given a school) and punishments (vilification, withdrawal of 
resources and parents withdrawing or placing their child(ren) elsewhere). A more sophisticated 
model would be of continuous active dialogue between schools and their stakeholders but 
unfortunately, this might misrepresent reality for many schools. 

7,4: The application of contingency theory has been made possible as the result of three 

significant trends in the development of school performance indicators: 

• the move away from purely descriptive statistics to those that can provide the basis for 
sound analysis; 

• the addition of contextual data; 

• the introduction and use of multilevel data which approximates educational activity at 
pupil, classroom and school level. 
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FIGURE 2: CONTEXT-INPUT-PROCESS-OUTCOME MODEL 
OF SCHOOLING (TAKEN FROM SCHEERENS, 1990, p.63). 



COM XX 

• achievemenr siiniuiants trom higher adminiscraUve levels 

• development of educational consumensm 

. "co-vaiiablcs” like school size- scudent-body composition, 
school category, uroan/rural 



i 



inputs 

. teacher experience 

- per pupil expenditure 

- parent support 




•V 





fprocejs ^ 

Uchooi Icvti I 

• degree of achievement oriented policy . 

I . educational leadership I 

I - consensus, cooperative planning of | 

I teachers t 

• quality of school curricula in terms of 

I content covered, and formal soucrure I 
l^rdcriy atmosphere ^ _| 

ZZ3cIZZZZZ 

fc/ossroo/n /eve/ i 

I - time*on*iask (including honwwork) 

• structured teaching 
|. opportunity to leant 
N high expectations of pupils* progress J 

• degree of evaluation and monitoring of 
I pupils* progress 

I ■ reinforcement j 



outputs 

student 

achievement, 
adjusted for: 

. previous 
achievement 
. intelligence 
.SES 
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FIGURE 4: ONE INTERPRETATION OF COMMUNICATION 
BETWEEN SCHOOLS AND THEIR STAKEHOLDERS. 



7.5: Research into Pis is definitely on the trajectory of more process oriented indices of 

performance. However, as Scheerens notes this 'could lead to evaluative conclusions of operation 
successful patient deceased', whereby all that is being achieved is the condition of monitoring 
each school, as if it was in an intensive care ward. To overcome this, he recommends anchoring 
process indicators to observable output indices of performances. This might be achieved by 
sophisticated, or not so sophisticated statistical techniques, or by ensuring they are fully 
grounded in a widely acceptable theory. It is tlirough the latter, that Schereens takes the 
tentative first steps towards valid 'process indicators'. Process indicators always refer to 
characteristics of the education system that can easily be manipulated. Schereens recommends 
their identification through interfacing of the findings of three distinct disciplines in educational 
research. These being:- 

• Sociological Discipline: this has tended to concentrate on the issues of (in)equality in 
education, when and where the neomarxist and feminist schools have predominated. 
Other schools within this discipline have concentrated on how best schools can serve 
society. The Coleman Report of 1966 falls into this discipline and is implicated as one of 
the major dialectic events of educational research, by imposing socio-economic class as 
the only factor that significantly affects, or for that matter effects, educational attainment. 
Findings from this field have tended to suggest that each school, as an entity of influence 
on its pupils' attainment, is essentially impotent. 

• Economics Discipline: has largely attempted to impose a linear input-output production 
model onto schooling outcomes. The only consistent predictor found being teacher 
experience (Hanushek, 1986). Findings suggest no definitive link with outputs measured 
in dollars or ECUs with educational outputs translated into the same currency. Only 
where and when there is greater variation in inputs does any causation or effect become 
apparent. 

• Psychological Discipline: this falls into two divisions: organisational behaviours as 
influenced by how a school's systems and procedures are structured and the social 
psychology of the classroom which is closely coupled with the instructional style of the 
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teacher. Mortimer et al (1988) essentially refuted the findings of the Coleman report by 
reporting that the organisation and ethos of a school does significantly contribute to pupil 
attainment when SES is statistically controlled. 

7.6: It is very alarming, but interesting to note that *In fact including contextual variables like 

student-body composition, school type, or national educational context can be seen as relatively neiu and 
very interesting development in school research' (Schereens, op cite) and this is, to some extent, a 
reflection of the research tools and methodologies available to educational researchers over the 
majority of this century. To some extent, researcher bias might have favoured inappropriate 
techniques and researcher expectations may have delivered the findings they sought. Such 
expectations and biases might explain the late emphasis on the whole picture and the 
consequential affects of magnifying individual processes and events in education out of all 
meaningful proportions. 

7.7: Through figure three, Scheerens demonstrates possibly only a fraction of all possible 

permutations or mechanisms involved in the delivery of education to each child. He also 
skilfully lists some of the possible links between meso (school) level factors and micro 
(classroom/ pupil) level factors and these are shown in Box Seven. As noted earlier schools can 
be effective in different ways according to the needs and aspirations of pupils and their parcitt^: 
Prescribing what effective schooling is, especially in process terms, would impose rigidities that 
would hinder the delivery of appropriate quality education: thus, amplifying the relevance of 
contingency theory to schooling. 



Box 7: 

• Structured teaching at classroom level can be simulated by means of explicit curricular policy at school level 
(e.g. school development plans); 

• 'high expectations of student performance’ is essentially a variable defined at teacher level, though its 
aggregate, an acliievemcnt oriented school policy, may be taken as a whole that is more than the sum of the 
parts; 

• order in the classroom will be enhanced by an orderly atmosphere in other parts of the school building; 

• frequent monitoring of pupils' progress will usually take place at classroom level, though this evaluation may 
be a result of a school evaluation policy and will benefit from instruments at this level, such as computerised 
school evaluation or a management information system; 

• opportunity to learn can be defined at the classroom level, but can also be seen as being enhanced by a school 

curriculum that is closely linked to educational objectives that determines the achievement tests. 



7.8: Scheerens also addressed the methodological issues that are critical to the adoption of 

such a model. The first being the measurement and quantification of behaviours, actions and 
events: it is never a straight forward transition from qualitative differences between school and 
classroom dynamics to objective statistical indices. Secondly, many of the studies on which to 
base the selection of school Pis suffer from severe methodological weaknesses. Therefore, 
imposing a purely positivist approach may not be possible and we may have to accept that the 
selection of performance indicators will always, to some extent, be an art; as Scheerens clearly 
states '... argument for an emergent school effectiveness model that is hard to neglect - though it is not 
very scientific - is its intuitive appeal' (Scheerens, 1991, p.393). Also, such a theory has to attempt 



to link school effectiveness research to more generally explanatory principles like learning 
theory, x-efficiency, and organisational theory. In spite of his cautionary notes, he sti" considers 
reviewing the literature as the best way forward for selecting Pis. 

7.9: School Pis according to Scheerens can be classified within what he refers to as a three 

dimensional framework. The three axis being:- 

1. Administrative level at which they are used. 

2. How the data is collected and aggregated. 

3. The level of inference within the measurements used. Qualitative indices often require 
greater understanding than quantitative indices. 



1 Issue 20: Invol ving teachers in the development of FIs. 

7.10: Climaco (1992) expands on Scheeren's concepts, in particular the use of Pis and their 
development as a mecharusm of positive organisational change within schools. Climaco and his 
colleagues, essentially developed Pis in partnership with Portuguese schools. Portugal since 
joining the European Union has placed considerable demands on its education system to meet 
the challenges of westernisation. Climaco cites Horowitz (1990) who advocates devolving the 
use of Pis to the lowest organisational level; in this instance the schools themselves, rather than 
central and local government. Climaco also segregates the concepts of indicator, information 
and performances and defines them as follows: 

Concept of Information: organisational intelligence crucial for the organisation's survival. 

Concept of Indicator: indicators are not merely statistics, but a piece of information relating 

to the actioning of objectives. 

Concept of Performance: essentially a multifaceted metaphor that implies a quality' result. It 

deals with the 'invisibilities' and visibilities of action and as such may 
be expressed by the results of action, as well as the conditions and 
contingencies under which action takes place, (p.327) 

7.11: Therefore, Climaco argues a PI is 'crucial information under the form of a statistical phrase, 
covering domains such as: student achievement and development, including cognitive and non-cognitive 
results; the contextual variables that affect the teaching and learning processes or the available resources 
and the use that is made of them.' (p.297) 

7.12: Climaco, in line with Horowitz's ideals, structured a project that facilitated schools in 
organising their own information systems and to explore the appropriateness of different 
potential performance indices. The project was democratised by teachers collating the necessary 
information themselves in a manner most appropriate for the contingencies within which their 
school operated. In effect the operationalisation was determined to considerable extent by how 
autocratic or democratic the school management was in each school. Institutional self-evaluation 
was achieved by asking school staff to reflect on the nature of Pis identifiers and their strength 
for determining the characteristics of educational quality. This approach produces what is 
referred to by Krashen (1982) as an 'affective filter' that is essentially an learning process in 
which common doubts and beliefs are confirmed projecting the school staff into a discovery 
process. This is essentially an iterative process in nature, allowing schools to identify where they 





needed to reform their education provision. Interestingly this approach yielded fifty indicators 
using the methodology outlined by Scheeren's model, highlighting the need for schools to select 
their <^,vn performance indicators that reflect local priorities or for that matter to chart 
individual schools' success at implementing desired organisational change. 

7.13: More significantly, the study indicated very few teachers had an overview of what their 
school was achieving, being only aware of the success of the education provision within their 
own departments. This may in part be due to schools not having the resources for managing 
and disseminating the necessary information for individual teachers to form such views. In 
addition, decision making schools tended not to be information based, i.e., reflect the 
circumstances of the school, but instead was norm referenced against other schools and what 
was perceived to be expected of the school. At the inception of the study, school staff tended to 
confuse the concept of an indicator with the metaphor of performance, but towards the end of 
the study, they recognised that an indicator can at best only encompass part of the performance 
of a school and this was in part due to a move away from a teacher centred view of their school 
performance to a more holistic and developmental interpretation. 



Issue 21: Educational Effectiveness as a holistic theoiy of education. 



7.14: Creemers (1993) has developed further a model of educational effectiveness, which draws 
on his joint work with Scheerens at the end of the 1980s (Scheerens and Creemers, 1989) and it 
was from this model that both Creemer's and Scheeren's models have been developed. Creemer 
first skilfully deconstructs the concept of educational effectiveness, drawing a distinction 
between it and school effectiveness. Educational effectiveness being a holistic theory that takes 
into account the outcomes of education, the inputs, the processes and contexts witliin which 
education takes place. School effectiveness studies has tended to concentrate on academic 
achievement and the definition and measurement of this has been criticised as poor. 
Educational effectiveness is linked to the achievement of "‘'•'r.tives, which distinguishes it from 
the study of school effects and the concept of school ^ which deals with the relationship 

between effects and inputs. Added value concept stresses tnat each pupil has a background, an 
aptitude for learning, a home environment and peer group that contributed to that pupil's skills 
and knowledge on entry to each stage in a child's educational career. Whereas educational 
effectiveness has to account for these background factors and link them to the achievement of 
objectives: this requires the measurement of specific abilities as well as performance measures in 
say maths and English. 



Issue 22: Since educational effectiveness is underpinned by objectives, it is essential to 
differentiate objectives from aspirations. 



1 



7.15: A crucial point raised by Creemer is that educational effectiveness is restricted to what 
schools can achieve and what they are for. It is useful at this point to make the distinction 
between aspirations and objectives: objectives address concentrate realities and project what 
might be achievable in the immediate and medium future and by doing so provide a 
mechanism for measuring school performance. Aspirations are often cited as objectives, thus 
causing anxieties amongst teachers, one example might be that all pupils leaving primary 
should be performing at National Curriculum level 3 to 4, when in fact in most inner-city 
schools, a sizeable minority might be performing at level 2 and only 25% might have reached 
level 4. An objective would be that most pupils should be performing between say levels 2.5 to 
3.5 on entry to secondary by 1996 /7. 
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Issue 23: Is it valid to use multiple criteria of effectiv 

7.16: Creemers secondly addresses the common argument in the literature that they should be 
multiple criteria for effectiveness and that schools should adopt as many as these as possible. 
The following multiple outcomes are cited by Creemer 

1. Acquisition of basic skills and knowledge. 

2. Compensation for inequity. 

3. Development of each child's social skills and attitudes. 

4. Development of higher order skills. 

5. The achievement of the 'new' educational objectives like creativity, moral behaviours, 
self-esteem and moral behaviours. 

Creemer critiques this approach, citing Treddlie and Stringfield (1993) who found that schools 
with low SES catchments with restricted objectives tended to do better than those with broad 
objectives. He also cites Van der Werf (1988) in support of his argument that schools that were 
the most innovative with educational goals do not achieve as well as less innovative schools: the 
more you zoant the less you get! Also, the misconception that quality is something different from 
equity in terms of the objectives encompassed with these concepts is correctly addressed by 
Creemers. To support his argument he cites Van der Werf and Weide (1991) who studied 
acquisition of Dutch by immigrant children and found that those pupils that attended schools 
which strive for excellence did better than Dutch pupils who attended less effective schools. 
Therefore, Creemers argues:- 

Going for quality can ... reduce the variance. Going for equity increases the variance (in 
pupil attainment). 

Again citing Van der Werf and Weide, Creemer expands his argument by stating that: (a) there 
are no differences between effective instruction and specific objectives with respect to quality 
and equity measures; and (b) activities related to effective instruction count for both high equity 
and quality. They are probably not much difference between the two. This he argues 
undermines the Nutall et al (1989) argument that effective schools are differentially effective for 
different groups of schools and therefore, what is effective for bright pupils might be effective 
for disadvantaged pupils. This he feels will feed through into the academic self-concept of 
pupils and their self-esteem in general and cites recent Dutch studies into the effect of academic 
outcomes on affective outcomes, in terms of pupils' self regard; Knuver (1993) is cited in support 
of this argument: one should not place not much emphasis on affective {and may be also other 
outcomes) as separate independent outcomes of education, (p.4) 




Issue 24: Small statistical effects can have significant outcomes. 






7.17: Creemers accepts that the effects of effective schools are small in the statistical sense, 
within the region of ± 3% of the factors studied to date, but these have a considerable influence 
on the success of each pupil's educational career. Also, the effects of schools are inherently 
unstable as one might expect by an institution energised by the people who make up its staff 
and student body and its other stakeholders. 




[ Issue 25; Are achievement and attainment one and the same? 

7.18: Thus laying the ground for the development of his model. The distinction is made 
between achievement, educational attainment and output. Achievement being the result of 
student background, abilities, tenacity and aptitude in conjunction with other inputs like school 
resources, which includes teachers' backgrounds, experiences and expectations. Educational 
attainment being the ultimate output, the educational or professional career of the pupil on 
leaving and output, or more specifically immediate outputs relate to test results cormected to 
educabonal objectives and it is these that form the criterion for effectiveness. In essence, 
Creemers is returning to what is achievable by schools - schools do not place pupils into careers, 
but they can influence GCSE results. 




Issue 26: The crucial factor is the amount of time pupils spend on task. 




7.19: Creemers agues that 'time-on-task' is a valuable index of school effectiveness as this is to 
some extent determined by processes at school and instructional level and can be extended by a 
school homework policy, but it has to be filled by opportunities to learn, which concerns the 
supply of learning materials, experiences, and exercises to acquire knowledge and skill. In sum, 
learning opportunities afforded by time-on-task are operalisation of education and tests and 
examinations are evaluative operationalisations of the same objectives. Through a meta-analysis 
of the literature, Creemers identifies three factors and these are listed in Box Eight. 



Issue 27: School organisation and policy are not the 
same and have different effects on school climate. 



7.20: Creemers advocates that many of the characteristics outlined in Box Eight have to be 
more tightly defined and the teacher is the central component because he or she operates at 
classroom level. They, essentially, are the ones who provide the learning materials, organise 
grouping procedures and produce differentiated work for pupils. At school level, Creemer 
makes the distinction between educational policy that is expressed as school development plans 
for example, and the organisation of the school that influences what occurs in the classroom. The 
educational policy of a school is codified as the school development plan and deals with the 
school's aims and objectives and Creemer reiterates the need for a linaited number of objectives. 
By referring to the concept of the organisation of the school, Creemer is referring to how the 
school meets its objectives and crucially the notion that school policy in respect to education is 
carried out by teachers and pupils. Creemer also refers to the concept of school climate in his 
model, which has to do with the responsibility teachers take for their pupils' progress and 
likewise the responsibility pupils take for their own learning and this is usually expressed as a 
quiet and orderly atmosphere. This is achieved through two mechanisms; good educational 
leadership of the school head; but also by the cohesion of the team and the control of students 
and teachers. The following factors at school level are therefore important according to 
Creemers model of school effectiveness:- 

1. Consistency: in the classroom, between textbooks and handouts, teacher behaviour and 

grouping procedures. 

2. Cohesion: every team member underlines the school's principles and behaves 

accordingly. 

3. Constancy: pupils receive the same treatment throughout their whole school career. 



4. Control: of teacher and pupils. 



Box 8: 

Learning Materials. 

1. Extent to which the curriculum offers opportunity to learn: quantity of subject matter, degree of 
overlap between goals that should be tested and subject matter. 

2. Explicitness and ordering of goals. 

3. Structuring and ordering of goals. 

4. Use of advanced organisers. 

5. Extent to which the curriculum evaluates student achievement. 

Grouping Procedures. 

1. Mastery learning, heterogeneous grouping and co-operative learning can induce higher 
effectiveness. 

2. The effectiveness is dependent on availability of differentiated learning material and testing, 
feedback and corrective measures. 

Teacher Behaviour. 

1. Management of the classroom. 

2. Orderly and quiet atmosphere. 

3. High expectations. 

4. Clear goal setting, including restrictive set of objectives, emphasis on basic skills, cognitive learning 
and transfer. 

5. Structuring the content; including sequence of objectives and content, advanced organiser, making 
use of prior knowledge of students and immediate exercises after presentation of new content. 

6. High and low order questions and appropriate wait time for answers. 

7. Corrective measures. 

8. Pacing. 



Above school level in Creemers' model is a whole spectrum of contextual variables like national 
and local education policies that dictate school policy. In many respects, Creemers' and 
Scheerens' models share common features and this would be expected since they share a 
common root (i.e., the tentative model put forward by Scheerens and Creemer in 1989). 
Creemers has decided however, to concentrate on time-on-task as the central factor in his 
school effectiveness model. 



8.0 



Conclusions. 



8.1: School performance indicators have always existed, but as of yet no holistic theory of 

education exists to assist in their selection. They are only effective when used in the context of 
school evaluation models and we also have to accept that schools have to operate within 
nebulous boundaries of performance that possibly explain why no official definitions of 
•• effective schools are readily available. It has been argued that the best judges of how effective 
individual schools are might be support staff, who are able to make unbiased judgements in situ 
by observing day-to-day events. Often in the literature and in debates the actual clients and 
providers of the education system are merely seen as a currency that permeate schools essential 
for their survival rather than as human beings. It has also been argued that good schools have 
well defined principles based on exhaustive debates and the educational leadership by the Head 
is instrumental to this occurring. The translation of whole school policies by department heads 
have a significant effect on the effectiveness of schools and therefore, it may be more 
appropriate to evaluate at departmental level. 

8.2: The overall aim may be defined as achieving equity through excellence. Excellence is only 

a comparative concept and the definition of equity has not always been stable and has 
progressed from compulsory education for all, through equal allocation of resources to the most 
appropriate mixture of resources for each child. The latter is mediated by parental choice, which 
can also be referred to as 'parental engineering'. 

8.3: Schools are evaluated by a diversity of agencies and attempts to make evaluations 

coherent and consistent have not entirely been successful. In addition to 'hard data' relating to 
outcomes and inputs, school evaluators also collate qualitative data relating to school processes. 
This qualitative data does not easily lend itself to meta-evaluation of school effectiveness and 
therefore, there is the need to produce indices that summarise this data. 

8.4: Returning to the concept of school effectiveness, there was the need to clarify what this 

concept means. Essentially it can have two operational definitions: first, one can define it within 
the context of free- market economics; second, it can be define by reference to concept of equity. 
It was assumed that the free market metaphor could be easily transferred to education, but this 
requires parents to demonstrate a high degree of 'educational literacy' and the physical rigidities 
of systeiTi, such as geography and buildings prevent a genuine free-market taking place in 
education. In many respects, 'League Tables' which were designed to facilitate a free market 
have failed to achieve this purpose, because they do not provide sufficient information for 
informed parental choice. 

8.5: Linked closely to the issues of the 'League Tables' are the issues of pupil assessment. 

Teachers tend to favour criterion-referenced-tests that do not easily translate into comparative 
data often sought by parents, who wish to know how tlieir child is getting along in comparison 
to other children of the same age. Criterion referenced tests when properly designed have a 
tendency to increase paperwork, as demonstrated by the implementation of teacher based 
National Curriculum assessment. A more sophisticated approach to school evaluation is 
required, which allows informed parental choice. Such evaluation would address learning 
environment and wpuld take into account the mixture of pupils in each schools and address 
issues such as their needs, aspirations, motivation and the appropriateness of educational 
provision. 

8.6: Concerns have been raised by Helsby and Smith for example, about the affect of Pis on 

teacher professionalism, partly as a reaction to the ascendancy of the empiricist model of 
accountability over the traditional subjective-impressionistic model of school evaluation. Their 
concerns fall into two categories: (1) the use of Pis deprofessionalises teachers as they will 
effectively become reactive automatons; (2) Pis are purely summative in nature. As 
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demonstrated by recent advances of school effectiveness, school performance indicators that are 
based on teaching processes, which are clearly linked to school outcomes, can provide the basis 
for organisational growth. Pis should be situated within the context of what Berg (1993) refers 
to as 'Developing Organisations' (DO), which is distinct from organisational development (OD). 
DO is involved with altering the culture within schools to facilitate greater effectiveness, 
whereas OD often involves radical change in organisational goals and structures to facilitate 
increased profits and is therefore more appropriate for commercial organisations. 

8.7: Comprehensive models of school effectiveness are still in their formative stages and the 

affects and interaction of individual variables have yet to be established. The appropriateness of 
research methods employed to determine these interactions in the past have been questioned 
and this makes model development more intuitive than empirical in nature. Therefore, there is 
the need to establish groupings of schools that are similar in academic performance to establish 
common patterns within them and to contrast them against other classes of schools. There is 
also the need to classify the areas that schools draw their intake from on the basis of common 
socio-economic features, as this will allow some manipulation of SES factors in future model 
building. 
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